\input epsf
\documentstyle[epsf]{article} 

\hyphenation{se-cre-tas}
\hyphenation{in-fra-es-tru-tu-ras}
\hyphenation{ex-pe-ri-men-tais}
\hyphenation{rou-ting}
\hyphenation{exis-ten-tes}
\hyphenation{ins-ti-tui-cao}
\hyphenation{ser-vi-dor ser-vi-do-res}
\hyphenation{a-pa-dri-nha-men-to}
\hyphenation{e-le-ctro-ni-co}
\hyphenation{par-ti-ci-pan-tes}
\hyphenation{es-ta-be-le-ci-men-to}
\hyphenation{da-ta-ba-se}
\hyphenation{admi-nis-tra-tors}
\hyphenation{avoi-da-ble}
\hyphenation{de-le-ga-ted}
\hyphenation{prob-lems}


\newcommand{\bold}{\bf }

\begin{document}


\title{\Large Ddt -- Domain Debug Tools \\
A Package to Debug the DNS Tree}

\author{Jorge Fraz\~ao -- frazao@ul.pt \\
and J. Legatheaux Martins -- jalm@ul.pt
\\
Departamento de Inform\'atica da \\
Faculdade de Ci\^encias da Universidade de
Lisboa\thanks{Ddt is a joint development of the Informatics Department of the
Faculty of Sciences of the University of Lisbon and ``PUUG -- 
The Portuguese Unix Systems Users Group'',
and has received a grant from ``RCCN Network -- Portuguese National R\&D 
Network''}}

\date{\today \\
Report DI-FCUL-1992-04 \\
\vspace{1 cm}
This paper has been proposed to a conference \\
Do not reproduce without
explicit authorization from the authors}


\maketitle

\section{Introduction}
\label{introd}

The domain name system (DNS) is a hierarchical, distributed naming system,
designed to address the problem of explosive growth of the number of 
machines in the current Internet environment.
It can be seen as some sort of hierarchical, distributed database, managed
in a decentralized way, by thousands of administrators.
The database is partitioned in zones. Each zone is replicated
by at least two servers, in a primary/secondary or master/slave scheme.
By its nature, organization and decentralized management style,
the introduction of errors and inconsistencies in this database
is not easily avoidable.
Those errors impact applications with more or less severity, leading
to resource wasting and/or service denial.
Unfortunately, there is no possibility to run a test over some sort
of ``global state'' of this database.

The problems described above lead us to develop the
{\em Ddt (Domain Debug Tools)\/}  package.
This package allows the verification of the coherence of
sub-sets of the database, both syntactically and semantically.
Most known errors are detected by Ddt.
Therefore, this package can play an important role
helping DNS administrators.
Besides this, it also allows the production of several statistics over the DNS data,
as for example hosts per domain, more popular names, etc.

The Ddt package is a set of commands
working over zone files, cached locally. We took this
option because it allows an efficient analisys of the coherence of 
a set of zones. This is the only way some errors can be detected.
Furthermore, it is much more efficient to work over local data than querying
servers on line. When the connections aren't good enough, as is in our 
situation, this is the only realistic alternative.
Zone files may be outdated when they are analyzed, however, even if
the servers were queried on line, it is also possible to face
a similar situation. This means that working over cached data is
really the only correct alternative.

In order to debug and test the effectiveness of Ddt, we have run a full test over
a sub-set of the DNS database, mostly concerning the European countries.
During the week from April 13th to April 20th we cached in our site almost all
the european zone files and ran an extensive analysis of this data.
This allowed us to obtain some conclusions about the most usual errors, as well as
some interesting results about the several ``cultures''
used by DNS administrators in the different countries, etc.
Furthermore, this analysis allowed us to discover several paradigmatic
errors. Some are simply ``curious or funny'', however, some are dangerous
and should be quickly corrected.  Some of those results take the form
of statistics tables. 

The reader should be aware of the fact that the examples shown are real ones.
Some of them have been corrected since the data used for the analysis has been
cached however, they were present in the european DNS the night of
the 14th April 1992.

This paper presents our work and the results obtained during the analysis
of the european DNS data. Sections 2 and 3 present an introduction to the DNS as
well as the rational behind DNS administration and the reasons that provided
motivation for building Ddt. Section 4, named ``European DNS culture and
folklore'', presents the results we got while
analysing the european DNS data. Finally, section 5 closes the paper with some
remarks about the limits of Ddt as well as some open issues.

\section{The Domain Name System}

In any distributed environment, application programs use a name system to
find addresses corresponding to machine names or, stated in a more general
way, use name systems to find attributes associated with named
objects\cite{grapevine}. These
name systems are essential to allow several sorts of bindings to take place.

In early days of ARPANET, or in small local networks with a dozen machines,
a file containing a central database (like ``/etc/hosts'') can be
reproduced by hand in every machine to solve the problem. However, in the
current worldwide Internet, with hundreds of thousands machines, this
method is no longer viable.

The domain name system (DNS) is a hierarchical, distributed naming system,
designed to address the problem of explosive growth of the number of 
machines\cite{comer-resumido}.
It can be seen as some sort of hierarchical, distributed database, managed
in a decentralized way, by thousands of administrators. It is hierarchical
because the name space is structured like a tree. Each branch of the tree
is called a {\em (sub)domain}. The tree can have as many levels as needed.
It is distributed because the management of each domain can be delegated
to a new machine and administrator. Because local domain administrators control their
branch of the tree, they can add, delete or change names easily.

Figure 1 presents a small slice of the current DNS tree. The tree has a
{\em root} and several branches. A domain name is similar to an Unix\footnote{Unix
is a Registred Trade Mark of A.T.\&T. in the U.S.A and other countries}
file path name with components written from
left to right and separated by a ``.'' (dot). A full qualified domain name,
like ``dec4pt.puug.pt.'' is a path from a leaf up to the root
(``.''). In general, the root domain name may be omitted. In almost all
situations, ``dec4pt.puug.pt.'' denotes the same object as ``dec4pt.puug.pt''.

\begin{figure}[hc]
\epsffile{dns.eps}
\caption{A small slice of the current DNS tree}
\end{figure}
\vspace{0.5cm}

The DNS database contains attributes associated with names
\cite{rfc-1034}. These attributes
are called {\em resource records (RR)\/}. Several RRs can be associated with
a name. Each one denotes the value of one attribute of the named object.
There are several classes of RRs:  A RRs (Address) allowing the
association of an address with the name, NS  RRs (Name Server) allowing
the association of a pointer to a name server, MX RRs (Mail eXchanger)
allowing the introduction of a mail relay willing to accept mail for the name
\cite{rfc-974},
HINFO RRs (Host INFOrmation) allowing the introduction of information
about a host, TXT RRs (TeXT) allowing the
association of any comments with the name, CNAME RRs 
(Canonical NAME) allowing the 
introduction of aliases, etc.

Each branch of the tree is managed by a special program called a
{\em name server\/}\cite{bind}.
Below is shown a portion of a dump of the set of RRs associated with the
domain name ``puug.pt'' present in current the DNS database. The dump has been
obtained with the user level utility {\em nslookup} (``Name System Lookup'') when
querying the machine ``dec4pt.puug.pt''.

\small

\begin{table}[h]
\begin{verbatim}
 puug.pt.                       TXT   ``Portuguese Unix network''
 puug.pt.                       TXT   ``Secretariat: sec@puug.pt''
 puug.pt.                       MX    10   dec4pt.puug.pt
 puug.pt.                       MX    20   inesc.inesc.pt
 puug.pt.                       NS    dec4pt.puug.pt                
 puug.pt.                       NS    host.fct.unl.pt               
 puug.pt.                       NS    inesc.inesc.pt                
 dec4pt                         A     192.84.62.1
 col                            CNAME colombo.puug.pt
 colombo                        A     192.67.76.4
 colombo                        MX    20   dec4pt.puug.pt
 dec4pt                         HINFO DECsystem-3100  Ultrix-4.0
\end{verbatim}
\caption{Portion of a dump of the ``puug.pt'' domain}
\end{table}

\normalsize

The example shows the use of several RRs associating attribute values with
names. For example, MX RRs stating that the machine
``dec4pt.puug.pt'' is willing to receive mail addressed to
``user@puug.pt'' and to ``user@colombo\-.puug.pt'' and that
machine ``inesc.inesc.pt'' is also willing to receive mail addressed to
``user@puug.pt'' (however, machine ``dec4pt.puug.pt'' should be used
preferably),  NS RRs stating
the list of name servers knowing authoritative data (see below) about the domain
``puug.pt'', A RRs introducing the addresses of the machines
``dec4pt.puug.pt'' and ``colombo.puug.pt'', a HINFO RR introduction information
about the machine ``dec4pt'', etc.

Any server or utility can interface the DNS system by quering name servers about
RRs associated with a name\cite{rfc-1035}.
It is possible to query about any kind of
attributes or about a specific kind of RR. If the server doesn't know
the name, it answers with a list of name servers that can be
queried instead. Eventually, with the list of name servers
running authoritative for the root (see below).
This would allow the querying client to walk from the root until
the right branch of the tree.


\section{Managing Name Servers}

A simplified vision of the DNS tree would persuade the reader that each domain
is associated with a server. That is not true for several reasons:

\begin{itemize}
\item
A server can manage more than one domain -- in general, domains under
the same authority, like a domain and its descendants. This set of domains
managed under the same authority is called a {\em zone}. In fact, a
server can even manage more than one zone.
\item
If only one server knew about a certain zone, it would become a central point
of failure. Therefore, each zone is known by at least two servers -- a primary and a
secondary or backup server. These two or more servers run in a master/slave
fashion. The primary is the only server that can be updated.
Secondary servers transfer periodically the primary database. When
queried about a zone, primary and secondary servers respond with
{\em ``authoritative data''}.
\item
In order to speed up querying and to scale down communications cost, servers
and clients cache the previously obtained replies. These are called {\em ``non
authoritative answers''} and play a very important role in overall DNS
performance. Associated with RRs  there are 
{\em Time To Live Timers\/} or {\em TTL Timers\/}, stating the period
during which a cached RR may be trusted. After the TTL period, the RR must be purged from
the cache.
\end{itemize}

As it can be seen, after updates there are database inconsistencies among 
the server running primary and the servers running secondary for the zone.
There are also inconsistencies with the cached data. These inconsistencies
are more or less tolerated by the different application level servers
and by users querying the DNS.

Each zone (set of domains managed under the same authority) is managed
by an individual, the {\em zone administrator}
\cite{rfc-1033}. The administrator is responsible
for the correct set up of the zone. Setting up a zone database involves
the definition of the zone itself and the introduction of the different RRs
belonging to it. 
When a server delegates its authority about a sub-zone to a server belonging
to one of its sub-zones,
{\em glue resource records} must be inserted. These special RRs associate
attributes with {\bold names outside the zone}, introducing non authoritative
data needed to allow a server querying
about the sub-zone, to be informed about the set of servers
knowing it (and their addresses), otherwise it would be impossible to get
the sub-zone servers addresses.
Moreover, each zone must be set up with a set of parameters
about the periodicity of zone transfers from primary to secondary servers,
retry timers if the previous zone transfer has failed, and period of
validity of a zone if its primary server cannot be contacted during a long
period of time.
Each zone has also a version number. This number is used to avoid unnecessary
zone transfers. Version numbers must always increase otherwise secondary server updates
will stop.

It is rather easy to introduce errors in the DNS database. Here is a list of
some of the more frequent ones:
\begin{itemize}
\item
Incorrect timers, causing too much retries in case of communication problems,
too high network traffic, poorly managed caches, or secondary servers
with out of date authoritative data during periods longer than reasonable.
\item 
Wrong version numbers introducing avoidable inconsistencies in the database.
\item
Incorrect or out of date glue records, leading to incorrect answers to
queries and to the spreading and caching of wrong RRs.
\item
Incorrect, illegal or too expensive mail routing policies.
\item
Inconsistencies between the set of servers running authoritative for a zone 
and the actual set of real servers.
\item
All sorts of misuse of the different types of RRs caused by several misunderstandings
of their purpose.
\item
Incorrect and non existent names introduced by a special abbreviation feature
of the server software (bind\/). This error is so frequent and dangerous
that it has been named the {\em ``missing trailing dot problem''}.
\item
Aliases pointing to nowhere, etc.
\end{itemize}

The introduction of all or some of these errors is not easily avoidable because
there are hundreds or even thousands of different administrators involved.
These administrators tend to be less and less skilled as long as we walk
far from the root domain or at least far from the {\em ``top level domains''}
(domains like ``.edu'', ``.com'', ``.mil'', and country domains), or when
their countries have recently connected to the Internet, like our own.
However, these errors are not easily detectable because the distributed
and decentralized nature of the DNS system, as well as its ability
to survive to wrong and inconsistent data, allows the system to continue
to produce useful work even in the presence of errors. Moreover, there is no
possibility to run a test over some sort of ``global state'' of the database.

Generally, when the administrator of a zone delegates authority
over a delegated zone (a descendent new branch of the tree), he or she tries,
as much as ``his or her time'' allows it, to analyze the set up of the delegated
name server. However, it is very difficult to do it in a systematic way and,
as soon as the delegated server appears to be well managed, it is up to its
administrator to take care of its future. Errors will only show up in
extreme strange behaved situations, when users complain or by manual
inspection. We have been deeply involved with the set up of the top level
domain for Portugal (``pt.'') and we are responsible for delegating new
zones under this domain, that's why the idea of building a tool to help
DNS administration was born.

\section{The Ddt Package}

\subsection{Goals and Presentation}

The problems described above lead us to the development of the
{\em Ddt (Domain Debug Tools)\/}  package.
This package allows the verification of the coherence of one or
more zones, both syntactically and semantically.
Most of the errors presented above are detected by Ddt.
Therefore, this package can play an important role
helping administrators to 
eliminate most of the usual errors.
Besides this, it also allows the production of several statistics over the DNS data,
as for example: hosts per domain, more popular names, etc.

The Ddt package is a set of commands
working over zone files cached locally. We took this
option because it allows an efficient analisys of the coherence of 
a set of zones. This is the only way some errors can be detected.
Furthermore, it is much more efficient to work over local data than querying
servers on line. When the connections aren't good enough, as is the
case in our country, this is the only realistic alternative.
Zone files may be outdated when they are analyzed, however, even if
the servers were queried on line, it is also possible to face
a similar situation. This means that working over cached data is
really the only alternative.

A command is available to try to cache the ``real world'' situation
as close as possible.
This command,
{\em ddt-xfer}
\footnote{{\em ddt-xfer} is a slightly modified version of the bind
{\em named-xfer} command.},
allows the transfer of a zone to the cache, and, eventually, 
all its descendants zones. If you have enough disk and bandwith,
you can cache all the zones of the world with one only command.
Probably, when you will end, you should start again because some of
the cached zones should not reflect the real ones anymore.
({\em ddt-xfer} would avoid unnecessary zone transfers by comparing
cached version numbers with the current ones.)

Ddt has commands to analyze zone definitions, zone RRs, zone
glue records, zone MX RRs, reverse mappings and zone statistics.
Each command can run over one or more zone files, allowing the analysis
of one zone or a set of zones as a whole, as for example, a country, or even a continent.

With the purpose of helping the interpretation of the diagnosis messages,
we have introduced four severity levels. With these levels we want to distinguish
between the most and the least important messages. The four levels are the
following:

\begin{itemize}
\item
{[Warning 1]} --- You should investigate it now!
\item
{[Warning 2]} --- Don't forget this warning!
\item
{[Warning 3]} --- You should analyze it someday!
\item
{[Comment]} --- You can ignore it / or Ddt has not enough data to take 
an effective decision.
\end{itemize}

Messages belonging to [Warning 1] level are the most serious. On the other
hand, messages belonging to [Comment] level are less important, but
this doesn't mean that these messages can be completely ignored.
Unfortunately,  there are some serious
errors that were included in the this level because, when the message
is displayed, Ddt has not enough
data (zone files) to take an effective decision. The next section presents
the available commands.

\subsection{Available commands}

You should start caching zones with the command {\em ddt-xfer}.
This command allows pulling zone files as we have already presented
above.
In the example below, the zone ``xx.com'' is pulled to the local cache
if and only if the version number defined in the file ``xx.com''
(in the current directory) and the true version number don't match.
If the server ``server.xx.com'' is not available, the server ``server.yy.com''
is queried.

\small

\begin{verbatim}
    ddt-xfer -z xx.com -f . server.xx.com server.yy.com

\end{verbatim}

\normalsize

To pull all the sub-zones of ``xx.com'', the ``-r'' parameter
should also be included:

\small

\begin{verbatim}
    ddt-xfer -z xx.com -f . -r server.xx.com server.yy.com

\end{verbatim}

\normalsize

After caching zone files, you  can start your analysis.
Start of authority RRs (SOA) are one of the most error prone and
dangerous RRs. These RRs introduce a zone and its parameters (mostly timers).
{\em Soac} is a command that analyzes SOA RRs (i.e., RRs 
describing the top node of a zone).
It verifies if the timers of the SOA RR are consistent, i.e., if they are 
inside an interval of values defined in the {\em .SOA-timers} file. 
It also verifies if the NS RRs are sound, i.e., if the listed servers
have authoritative answers for data belonging to the zone. Moreover,
it verifies version numbers matching.
The example shown below presents a situation where the refresh time is very low (6 minutes).
This kind of error is very usual and can cause unnecessary traffic.
The second message is more serious, because the ``server.xx.com'' should be an
authoritative server but it isn't.

\small

\begin{verbatim}
    $ soac xx.com
    SOA Record
    Refresh 360 [recommended value: 28800]
    Authoritative servers
    server.xx.com   [Not authoritative answer]

\end{verbatim}

\normalsize


The command {\em rrc} finds syntactic and semantic errors in the master file. 
Errors like names without trailing dot, invalid host names, etc., are caught. 
All the errors in RRs associated with names outside the zone are also included.
However, the messages related with this kind of errors got [Comment]
severity level because, during the analysis, the fact that those zones were also
analyzed is not recorded.
In the example below, a syntax error that is caused by the absence
of a dot in the master file is shown. This kind of error is very usual. The
second message warns about a RR where a host should be introduced
instead of its address ``192.0.0.1''.

There are also 
two messages about an undefined host name. The first one is more serious
because the name of the host belongs to the set of authoritative
data (i.e., inside the analyzed zone).

\small

\begin{verbatim}
    $ rrc xx.com
    Perharps name without trailing dot: host.xx.com.xx.com
    Hostname 192.0.0.1 should be in domain notation
    No A/CNAME record found for server.xx.com
    No A/CNAME record found for server.yy.com [Can't verify it]

\end{verbatim}

\normalsize

Glue records\footnote{A glue record is
a A RR of a server that doesn't belong to the zone. This server can be a
name server for the zone or for a delegated sub-zone} 
are a kind of RRs that is generally responsible for the presence of
most inconsistencies in the DNS, therefore Ddt supplies a command 
to check these RRs -- {\em grc}.
This command checks glue records in the zone file.
The glue record can be present in the transferred (cached) zone.
However, due to the behaviour of the bind transfer, it can not be
really in the true master file. Therefore,  the diagnosis messages
have only the purpose of warning the administrator for the possibility
of their existence. All glue
records found are separated in two kinds, necessary and unnecessary glue
records. When the glue record introduces the address of a server of a 
delegated domain and that server belongs to a delegated domain,
the corresponding glue record must be present.
Below, an example is shown with the two kinds of glue records: necessary and
unnecessary. The first one is necessary because it's needed to
``glue'' the zone with its delegated sub-zone, since the server is itself inside the
sub-zone.

\small

\begin{verbatim}
    $ grc xx.com
    Necessary glue records. Name servers of delegated domains.
    server.yy.xx.com   IN  A   192.1.1.1
    Unnecessary glue records. If present, should be removed.
    server.yy.com   IN  A   192.0.0.1

\end{verbatim}

\normalsize

As MX RRs are often misunderstood, and in spite of knowing
that it is impossible to  find a generic policy for mail routing
(this is sometimes more a matter of politics than of technics)
we have decided to include a command allowing its analysis in the Ddt package.
This command analyzes MX RRs present in the cached zone file.
Its only purpose is to highlight possible configuration errors.
Those warnings are considered as evidence of possible strange
mail routing policies in two cases. If there are MX RRs pointing to hosts
outside the zone with preference value less than those MX RRs pointing to 
internal hosts, a message is displayed.
If there are MX RRs pointing to hosts outside the zone,
a second message is also displayed.
The last case is a more normal one. The purpose of the warning
is the highlighting of the policy(ies) used in a region,
or in a country, etc.
In the following example we can show some examples of strange mail routing
policies.

\small

\begin{verbatim}
    $ mxc xx.com
    MX record pointing to an external host.
    xx.com  IN	MX  20	mailhost.yy.com
    MX record to local or delegated host after MX record to external host.
    yy.xx.com	IN MX	10  mailserver.yy.com

\end{verbatim}

\normalsize


Finally, we have introduced a command to verify reverse mappings -- {\em rmc}.
This command verifies if there are some PTR RRs binding an address with
a host name without an A RR binding the name with the same address. It also verifies
the opposite situation, i.e., it tests if there are any PTR RR for the pair host/address
defined in a A RR.  As can be seen below,
we can see that there is an A RR for host10.xx.com. with
the address 192.0.0.10 but the RR defining the reverse binding doesn't exist.
The second message indicates the opposite situation, i.e., a PTR RR is present,
but the A RR for same host/address isn't.

\small

\begin{verbatim}

    $ rmc xx.com 0.0.192.in-addr.arpa
    ## XX.COM. ##
    No 10.0.0.192.in-addr.arpa PTR RR found for host10.xx.com.

    ## 0.0.192.IN-ADDR.ARPA. ##
    No host5.xx.com. A RR found with 192.0.0.5

\end{verbatim}

\normalsize


\subsection{Statistic scripts}

In order to help gathering DNS statistics, Ddt includes also several filter scripts.
These scripts allow the production of statistics over the zone files, thus it is
possible to get the number of hosts per domain, the number of hosts per net, 
more popular names, etc. 
To obtain statistics not foreseen with the actual scripts, it's easy
to build more scripts, with {\em awk}, {\em grep}, {\em sort}, etc., 
Unix commands.
The following command line returns the most popular names in Portugal:

\small

\begin{verbatim}
    expand /usr/local/ddt.cache/pt/* | host | names-stat | sort -rn

\end{verbatim}

\normalsize


The ``expand'' command replaces all the relative names by the full qualified 
ones, the ``host'' command filters the host names and finally the ``names-stat'' 
command counts the repeated names.

The next command line generates the statistics about the number of hosts per net in Italy:

\small

\begin{verbatim}
    expand /usr/local/ddt.cache/it/* | hosts-addr | nets | sort -rn

\end{verbatim}

\normalsize


The ``hosts-addr'' command filters host names with the its ip address 
and the ``nets'' command counts the number of hosts per net.



\section{European DNS Culture and Folklore}

During the week from April 13th to April 20th we cached in our site almost all
the european zone files and ran an extensive analysis of this data.
This allowed us to obtain some conclusions about the most usual errors, as well as
some interesting results about the several ``cultures''
used by DNS administrators in the different countries, etc.
In this section we are going to present those results as well as
some of the most ``curious'' or more
meaningful facts we faced during our analysis. Some of those
results take the form of statistics tables.

The results presented show the ``state of the european DNS'' in the night of April 13th to
14th.

The analysis shows that, generally, DNS administrators follow
the options taken by the top level domain administrator of their own country or
by the administrators of the most known domains. This proves that
there is a path for knowledge dissemination among administrators leading
to the dissemination of ``regional'' habits. For example, it is possible
to use several management policies to guarantee the serial number growth (See Table~4). 
In general, this option is relatively uniform within the same country.

Another curious fact is related with SOA timers.
In average south European countries are using higher values.
This probably means that price and quality of
communications in those countries have been worse than in the
north European countries.

As it is generally known, Germany is the champion in the size of host names.
The first 200 longer european host names are all below the ``de'' domain!
Curiously, Italians win in the shortest host name (``it'' with the address
131.114.1.30).

Several information, like number of hosts, domains, zones, etc. per country 
are presented in Table~2. The first three columns are respectively
number of zones, domains and hosts per country. The fourth is the
total space used to store zone files. Fifth, sixth and seventh columns
combine the first three ones, i.e., average domains per zone, average hosts per zone and
average kbytes per zone file 
for each country. The two last columns indicate the average of zone names 
depth and depth of the biggest zone name.

\small

\begin{table}[htb]
\begin{center}

\begin{tabular} {|l|r|r|r|r|r|r|r|r|r|}\hline
 & \tt Zones & \tt Domains & \tt Hosts & \tt Kbytes & \tt Domains & \tt Hosts & \tt Kbytes & \multicolumn{2}{c|}{\tt Depth-Zone}\\ \cline{9-10}
 & & & & & \multicolumn{1}{c|}{\tt Zone} & \multicolumn{1}{c|}{\tt Zone} & \multicolumn{1}{c|}{\tt Zone} & \tt Avr & \tt Max\\ \hline 
\tt AT & \tt 99 & \tt 150 & \tt 4345 & \tt 459 & \tt 1.5 & \tt 44 & \tt 4.6 & \tt 3.65 & \tt 5\\
\tt BE & \tt 15 & \tt 187 & \tt 1043 & \tt 133 & \tt 12.5 & \tt 70 & \tt 8.9 & \tt 3.53 & \tt 4\\
\tt BG & \tt 1 & \tt 7 & \tt 0 & \tt 3 & \tt 7.0 & \tt 0 & \tt 3.0 & \tt 1.00 & \tt 1\\
\tt CH & \tt 37 & \tt 218 & \tt 13367 & \tt 1143 & \tt 5.9 & \tt 361 & \tt 30.9 & \tt 2.00 & \tt 3\\
\tt CS & \tt 4 & \tt 32 & \tt 177 & \tt 15 & \tt 8.0 & \tt 44 & \tt 3.8 & \tt 2.00 & \tt 3\\
\tt DE & \tt 651 & \tt 2316 & \tt 38376 & \tt 4737 & \tt 3.6 & \tt 59 & \tt 7.3 & \tt 3.00 & \tt 5\\
\tt DK & \tt 64 & \tt 241 & \tt 2319 & \tt 353 & \tt 3.8 & \tt 36 & \tt 5.5 & \tt 2.38 & \tt 4\\
\tt ES & \tt 120 & \tt 393 & \tt 3086 & \tt 330 & \tt 3.3 & \tt 26 & \tt 2.8 & \tt 2.05 & \tt 3\\
\tt FI & \tt 108 & \tt 346 & \tt 14493 & \tt 1147 & \tt 3.2 & \tt 134 & \tt 10.6 & \tt 2.56 & \tt 4\\
\tt FR & \tt 254 & \tt 427 & \tt 16024 & \tt 1643 & \tt 1.7 & \tt 63 & \tt 6.5 & \tt 2.48 & \tt 4\\
\tt GR & \tt 41 & \tt 49 & \tt 541 & \tt 109 & \tt 1.2 & \tt 13 & \tt 2.7 & \tt 2.51 & \tt 3\\
\tt HU & \tt 2 & \tt 8 & \tt 5 & \tt 4 & \tt 4.0 & \tt 2 & \tt 2.0 & \tt 2.00 & \tt 3\\
\tt IE & \tt 16 & \tt 64 & \tt 515 & \tt 87 & \tt 4.0 & \tt 32 & \tt 5.4 & \tt 2.31 & \tt 3\\
\tt IL & \tt 26 & \tt 42 & \tt 2552 & \tt 148 & \tt 1.6 & \tt 98 & \tt 5.7 & \tt 3.31 & \tt 4\\
\tt IS & \tt 6 & \tt 10 & \tt 285 & \tt 22 & \tt 1.7 & \tt 48 & \tt 3.7 & \tt 2.33 & \tt 3\\
\tt IT & \tt 127 & \tt 317 & \tt 3947 & \tt 1627 & \tt 2.5 & \tt 31 & \tt 12.8 & \tt 2.88 & \tt 4\\
\tt NL & \tt 230 & \tt 529 & \tt 14583 & \tt 1578 & \tt 2.3 & \tt 63 & \tt 6.9 & \tt 2.94 & \tt 4\\
\tt NO & \tt 243 & \tt 357 & \tt 12808 &  \tt 1037 & \tt 1.5 & \tt 53 & \tt 4.3 & \tt 2.65 & \tt 4\\
\tt PL & \tt 31 & \tt 37 & \tt 324 & \tt 74 & \tt 1.2 & \tt 10 & \tt 2.4 & \tt 2.94 & \tt 4\\
\tt PT & \tt 43 & \tt 48 & \tt 1158 & \tt 114 & \tt 1.1 & \tt 27 & \tt 2.7 & \tt 2.31 & \tt 3\\
\tt SE & \tt 262 & \tt 1803 & \tt 18836 & \tt 2200 & \tt 6.9 & \tt 72 & \tt 8.4 & \tt 2.79 & \tt 6\\
\tt SU & \tt 1 & \tt 105 & \tt 0 & \tt 12 & \tt 105.0 & \tt 0 & \tt 12.0 & \tt 1.00 & \tt 1\\
\tt TN & \tt 2 & \tt 3 & \tt 8 & \tt 6 & \tt 1.5 & \tt 4 & \tt 3.0 & \tt 1.50 & \tt 2\\
\tt UK & \tt 191 & \tt 1027 & \tt 24371 & \tt 2463 & \tt 5.4 & \tt 128 & \tt 12.9 & \tt 3.60 & \tt 5\\
\tt YU & \tt 2 & \tt 18 & \tt 13 & \tt 7 & \tt 9.0 & \tt 6 & \tt 3.5 & \tt 1.50 & \tt 2\\ \hline
\end{tabular}

\end{center}
\caption{Several data per country  (1992, April the 14th)}
\end{table}
\normalsize

Next sections present several more results for some other aspects.


\subsection{The structure of the name tree}

Four countries in Europe have choosen to have domains
like AC, CO, GOV, etc. below the country top level domain: Austria, Belgium, 
Israel and United Kingdom.
All the others have choosen to have organization names below the top
level domain.
Another interesting result is the depth of the SOA records owner
(name), minimum and maximum
is respectively 1 for Bulgaria and former Soviet Union, and 6 for
Sweden (``adm.hj.se.adm.hj.se'', ``tekn.hj.se.tekn.hj.se'' and 
``uki.hj.se.uki.hj.se'' -- this seems to be a missing trailing dot!).

\subsection{Timers}

As we have already presented, southern countries use higher timers.
Table~3 brings together the medium timer values used in each country.


\small

\begin{table}[htb]
\begin{center}

\begin{tabular}{|l|r|r|r|r|} \hline
 & \multicolumn{1}{|c|}{\tt \ \ Refresh\ \ } & \multicolumn{1}{|c|}{\tt \ \ \ Retry\ \ \ } & \multicolumn{1}{|c|}{\tt \ \ Expire\ \ \ } & \multicolumn{1}{|c|}{\tt Default TTL}\\ \hline
\tt AT & \tt 21058 & \tt 3074 & \tt 708541 & \tt 69927\\ 
\tt BE & \tt 6720 & \tt 1480 & \tt 2528640 & \tt 195840\\
\tt BG & \tt 86400 & \tt 14400 & \tt 2592000 & \tt 345600\\
\tt CH & \tt 21873 & \tt 4092 & \tt 1479989 & \tt 10657\\
\tt CS & \tt 67500 & \tt 26100 & \tt 10656000 & \tt 280800\\
\tt DE & \tt 22227 & \tt 2663 & \tt 2427647 & \tt 95146\\
\tt DK & \tt 11095 & \tt 3976 & \tt 3526706 & \tt 77634\\
\tt ES & \tt 28050 & \tt 6840 & \tt 677040 & \tt 88200\\
\tt FI & \tt 18300 & \tt 1569 & \tt 1983267 & \tt 44444\\
\tt FR & \tt 22632 & \tt 4886 & \tt 3381676 & \tt 166468\\
\tt GR & \tt 38371 & \tt 13580 & \tt 2937600 & \tt 198088\\
\tt HU & \tt 86400 & \tt 9000 & \tt 3096000 & \tt 216000\\
\tt IE & \tt 41625 & \tt 6825 & \tt 1150200 & \tt 153787\\
\tt IL & \tt 30750 & \tt 7502 & \tt 2666215 & \tt 115477\\
\tt IS & \tt 28800 & \tt 7200 & \tt 604800  & \tt 86400\\
\tt IT & \tt 42553 & \tt 2489 & \tt 1698208 & \tt 132718\\
\tt NL & \tt 20413 & \tt 6098 & \tt 1028871 & \tt 84346\\
\tt NO & \tt 11956 & \tt 1260 & \tt 1347906 & \tt 76274\\
\tt PL & \tt 69213 & \tt 6155 & \tt 2508387 & \tt 195097\\
\tt PT & \tt 72686 & \tt 7200 & \tt 1357714 & \tt 140914\\
\tt SE & \tt 10538 & \tt 1773 & \tt 2668534 & \tt 69703\\
\tt SU & \tt 86400 & \tt 14400 & \tt 2592000 & \tt 345600\\
\tt TN & \tt 50400 & \tt 9000 & \tt 734400 & \tt 129600\\
\tt UK & \tt 12187 & \tt 3777 & \tt 1725540 & \tt 80915\\
\tt YU & \tt 28800 & \tt 7200 & \tt 604800 & \tt 86400\\ \hline
\end{tabular}
 
\end{center}
\caption{Medium SOA timers per country  (1992, April the 14th)}
\end{table}

\normalsize

More interesting, and more serious too, are admnistrators that were using values
clearly inadequate as for example:
\begin{itemize}
\item
Expire above 1 year (36000000 seconds);
\item
The four timers with 15 seconds (``silab.dsi.uniti.it'');
\item
Null default TTL (``agw.docs.uu.se'').
\end{itemize}


\subsection{Serial numbers}

Table~4 presents the dominant policies used in each country for
insuring that serial (version) numbers are always growing. We can find three strategies for
serial number management: only incremental, starting with 1 and incrementing
1 by 1; in dotted decimal notation, i.e., the serial number is in the following
format: Release.Version (for example 1.01); and date codification
(with several variants: using 2 or 4
digits on the letf side for the year, and using 0 or more digits on the right
side for the version in the same day).


\small

\begin{table}

\begin{minipage}{14cm}
\begin{center}
\begin{tabular} {|l|c|c|c|} \hline
 & \multicolumn{1}{|c|}{\tt Only} & \multicolumn{1}{|c|}{\tt Dotted} & \multicolumn{1}{|c|}{\tt Date}\\
 & \multicolumn{1}{|c|}{\tt incremental\ } & \multicolumn{1}{|c|}{\tt \ \ \ decimal\ \ \ } & \multicolumn{1}{|c|}{\tt codification}\\ \hline 
\tt AT & \tt many & \tt few & \tt few\\
\tt BE & \tt none & \tt few & \tt many\footnote{The format most used is YYMMDDV.}\\
\tt BG & \tt none & \tt all & \tt none\\
\tt CH & \tt many & \tt some & \tt few\\
\tt CS & \tt some & \tt some & \tt some\\
\tt DE & \tt some & \tt some & \tt many\footnote{Several different formats are used and there are some zones that join the number 53 (port of domain service) at the left of the date format.}\\
\tt DK & \tt few & \tt some & \tt many\\
\tt ES & \tt few & \tt none & \tt many\footnote{Almost all use the format YYMMDDVVV.}\\
\tt FI & \tt few & \tt few & \tt many\footnote{The format most used is YYYYMMDDVV.}\\
\tt FR & \tt few & \tt few & \tt many\footnote{The following two formats: YYMMDDVV and YYMMDDV, are the most used ones.}\\
\tt GR & \tt some & \tt many & \tt none\\
\tt HU & \tt none & \tt some & \tt some\\
\tt IE & \tt some & \tt none & \tt some\\
\tt IL & \tt some & \tt few & \tt none\\
\tt IS & \tt none & \tt none & \tt all\footnote{All zones use the format YYYYMMDDVV.}\\
\tt IT & \tt some & \tt few & \tt many\footnote{The format most used is YYMMDD.}\\
\tt NL & \tt some & \tt some & \tt many\\
\tt NO & \tt few & \tt many\footnote{There are many serial numbers with six zeros between the number and the version (ex. 100000015).} & \tt few\\
\tt PL & \tt few & \tt few & \tt many\footnote{The format most used is YYMMDDVV.}\\
\tt PT & \tt some & \tt none & \tt many\\
\tt SE & \tt few & \tt few & \tt many\footnote{The format most used is YYYYMMDDVV.}\\
\tt SU & \tt none & \tt all & \tt none\\
\tt TN & \tt none & \tt all & \tt none\\
\tt UK & \tt few & \tt few & \tt many\footnote{The format most used is YYYYMMDVVV.}\\
\tt YU & \tt all & \tt none & \tt none\\ \hline
\end{tabular}
\caption{Policies used in the serial number make up  (1992, April the 14th)}
\vspace{1 cm}
\end{center}
\end{minipage}
\end{table}

\normalsize

There were some administrators that seemed to follow no
apparent heuristic, at least at first sight, like the following examples:
2147483647 in ``iiasa.ac.at'', 587000483 in ``epfl.ch'', etc.
The main reasons causing a regression of the 
serial number, we have caught, are:
\begin{itemize}
\item 
Change of the management strategy of the serial number.
\item
Wrong usage of a date to generate a serial number. There are two kinds of such
errors: using the
date in an incorrect order (50392 -- ``ariadne-t.gr'', 150192 -- 
``hitec.ariadne-t.gr'', 40292 -- ``nrcps.ariadne-t.gr'' all these serial
numbers have been apparently generated using a date in inverse order, DDMMYY);
another situation also very frequent is the usage of one  only digit when the day or month is
lower than 10, like the serial number of ``c3consult.comm.se'', 199112501.
\item
Bad manipulation of the serial numbers in dotted decimal notation. 
For example if a serial number 
with value 1.31 would be updated to a value like 1.4, this would cause a
regression, because the dot is replaced with 3 zeros by the bind server.
(the value 10004 is lower than 100031).
\end{itemize}

We also found some zones with a negative serial number.


\subsection{Name servers}

Just as it can be foreseen, we found frequent discrepancies between
the name servers list introduced in a zone and the real list of servers 
running authoritative for the same zone. 
One of the most curious cases was the zone ``tmt.tele.fi'' presenting ``.''
(root) as one of its secondaries !

In Table~5 we can find the top 20 most popular name servers.

\small

\begin{table}[htb]
\begin{center}

\begin{tabular}{r@{\hspace{0.1in}}l}
\tt 348 & \tt deins.informatik.uni-dortmund.de\\
\tt 261 & \tt layon.inria.fr\\
\tt 133 & \tt noc.belwue.de\\
\tt 118 & \tt sun.iris-dcp.es\\
\tt 92 & \tt goya.dit.upm.es\\
\tt 88 & \tt aun.uninett.no\\
\tt 86 & \tt ns.germany.eu.net\\
\tt 86 & \tt nac.no\\
\tt 72 & \tt columba.udac.uu.se\\
\tt 68 & \tt runix.runit.sintef.no\\
\tt 64 & \tt hydra.helsinki.fi\\
\tt 63 & \tt shiva.jussieu.fr\\
\tt 62 & \tt cendrillon.lptl.jussieu.fr\\
\tt 57 & \tt rs2.hrz.th-darmstadt.de\\
\tt 57 & \tt rs1.hrz.th-darmstadt.de\\
\tt 55 & \tt cancer.ucs.ed.ac.uk\\
\tt 54 & \tt dns.dcs.ed.ac.uk\\
\tt 52 & \tt xlab-0.ed.ac.uk\\
\tt 52 & \tt wilde.ucs.ed.ac.uk\\
\tt 52 & \tt pythia.edvz.univie.ac.at\\
\tt 52 & \tt directory.ed.ac.uk\\
\tt 52 & \tt alf.uib.no\\
\end{tabular}

\end{center}
\caption{Top 20 most popular name servers in Europe  (1992, April the 14th)}
\end{table}
\normalsize

\subsection{Using MX RRs}

MX RRs are really very useful! We found 130994 of these records in the 
european database.
Obviously, this represents a lot of different policies of mail routing and
we found lots and lots of MX RRs pointing to unknown hosts. France
is the country with less MX RRs pointing to hosts
outside the country (with 0 of such MXs). This doesn't mean that
all of those MXs were pointing to valid hosts. At least in 437 of these RRs
a missing trailing dot is present in the name of the mail gateway.

Finally, it is also interesting to know the top 20 most popular european gateways
(See Table~6).


\small

\begin{table}[htb]
\begin{center}

\begin{tabular}{r@{\hspace{0.1in}}l}
\tt 5627 & \tt mcsun.eu.net\\
\tt 5585 & \tt chenas.inria.fr\\
\tt 3800 & \tt dxmint.cern.ch\\
\tt 3654 & \tt cernvax.cern.ch\\
\tt 3266 & \tt sicvaxb.epfl.ch\\
\tt 3250 & \tt mail.germany.eu.net\\
\tt 2737 & \tt nsfnet-relay.ac.uk\\
\tt 2716 & \tt sun2.nsfnet-relay.ac.uk\\
\tt 2009 & \tt relay.surfnet.nl\\
\tt 1804 & \tt hearnvax.nic.surfnet.nl\\
\tt 1516 & \tt mailgzrz.tu-berlin.de\\
\tt 1419 & \tt mailszrz.zrz.tu-berlin.de\\
\tt 1379 & \tt nz11.rz.uni-karlsruhe.de\\
\tt 1379 & \tt netserv.rz.uni-karlsruhe.de\\
\tt 1335 & \tt sunic.sunet.se\\
\tt 1264 & \tt sun4nl.nluug.nl\\
\tt 1159 & \tt dutrun.tudelft.nl\\
\tt 1157 & \tt tudrva.tudelft.nl\\
\tt 1086 & \tt funet.fi\\
\tt 1059 & \tt chx400.switch.ch\\
\end{tabular}

\end{center}
\caption{Most popular mail gateways in Europe  (1992, April the 14th)}
\end{table}

\normalsize

\subsection{The missing trailing dot}

This is a very known and real problem. It seems that French colleagues were the
champions of ``forgetting the dot'' with at least 532 suspect names.
The absolute champion was the zone ``ircam.fr'' with 428 missing trailing
dot errors in its master file, 214 were in MXs pointing to 
``vaslav.ircam.fr.ircam.fr'' and the others 214 were also in MXs pointing
to ``nadia.ircam.fr.ircam.fr''.

One of the most interesting repercussion of this error
was the introduction of the zones
``adm.hj.se.adm.hj.se'', ``tekn.hj.se.tekn.hj.se'' and ``uki.hj.se.uki.hj.se''.


\subsection{Miscellaneous}

It is possible to find some funny names in the DNS.
One such case was one MX record owned by ``bibsys.unit.no'' pointing
to host ``taken.out.of.service.unit.no'' and the other was host name
``\#gnasher.cineca.it''. It seems that the ``cineca.it'' administrator
would want to comment this record in the master file (the comment caracter is
``;'', not ``\#'').

In Ireland, it seems that renaming  the ``IEUNet'' backbone was very
popular. Besides the true name ``dec4ie.ieunet.ie'', there were also the folowing names
in its DNS database: ``dec4ie.eu.net'' (with 19 references), 
``dec4ie.ieunet.net'' and ``dec41e.ieunet.ie'' (both with 1 reference).

One of the most frequent cause of inconsistencies in DNS are CNAME RRs defining
a local nickname for invalid hosts.

Some errors were caused by administrators that don't know what is
the real usage of some resource records. One example of this is the insistence of
using HINFO RRs to associate descriptive information to all type of
domain names, like the following we have found:
\small

\begin{verbatim}
uit.no.                         IN  HINFO   "University" "ofTromso"
localhost.rz.uni-karlsruhe.de.  IN  HINFO   "local" "local"
\end{verbatim}

\normalsize

In Table~7 we present the top 40 most popular host names in Europe.


\small

\begin{table}[htb]
\begin{center}

\begin{tabular}{r@{\hspace{0.1in}}lp{0.5in}r@{\hspace{0.1in}}l}
\tt 1168 & \tt localhost & & \tt 59 & \tt castor\\
\tt 117 & \tt venus & & \tt 57 & \tt sirius\\
\tt 102 & \tt pluto & & \tt 56 & \tt saturn\\
\tt 91 & \tt pc1 & & \tt 54 & \tt pc5\\
\tt 89 & \tt cisco & & \tt 52 & \tt xterm1\\
\tt 88 & \tt mars & & \tt 52 & \tt uranus\\
\tt 88 & \tt loopback & & \tt 52 & \tt mac2\\
\tt 88 & \tt iris & & \tt 51 & \tt pc6\\
\tt 85 & \tt pc2 & & \tt 51 & \tt helios\\
\tt 84 & \tt obelix & & \tt 50 & \tt titan\\
\tt 80 & \tt asterix & & \tt 50 & \tt gauss\\
\tt 79 & \tt idefix & & \tt 48 & \tt snoopy\\
\tt 75 & \tt jupiter & & \tt 48 & \tt bilbo\\
\tt 73 & \tt pc3 & & \tt 47 & \tt marvin\\
\tt 72 & \tt hermes & & \tt 47 & \tt mac1\\
\tt 71 & \tt zeus & & \tt 47 & \tt loghost\\
\tt 69 & \tt pollux & & \tt 47 & \tt frodo\\
\tt 67 & \tt orion & & \tt 47 & \tt donald\\
\tt 65 & \tt charon & & \tt 47 & \tt apollo\\
\tt 61 & \tt pc4 & & \tt 46 & \tt merlin\\
   & & & \tt 46 & \tt gandalf\\
\end{tabular}

\end{center}
\caption{Top 40 most popular host names in Europe (1992, April the 14th)}
\end{table}

\normalsize

As expected, ``pc1'', ``pc2'', ``pc3'', \dots , were very popular names. Curiously,
``mac2'' was more popular than ``mac1''. This is probably due to the fact
that the name ``mac2'' is related to a ``Macintosh\ II'' computer model.

Another important point was the presence of ``localhost'' as the most popular
host name in Europe and probably elsewhere. ``localhost'', as well as ``loopback'',
are reserved names allowing any site to designate itself and its network. These
names have the same role of ``self'' in the Smalltalk language for example. It is a general
mechanism allowing a site to call himself without even knowing its name. ``localhost''
is directly related with the reserved address 127.0.0.1 and ``loopback'' is
directly related with the network 127.0.0.0. These addresses denote a virtual host
(``this host'') and a virtual network (``this network''). The introduction of these
addresses in the DNS is meaningless except for providing the mapping of the
name ``localhost'' to the address 127.0.0.1 and its reverse mapping.

However, sometimes we found ``localhost'',
with or without the trailing dot as the right side of a MX or NS
record. Whenever a server queries about the address of the mail relay, or
the address of the name server, it can get, in these situations, the address
127.0.0.1, which means itself! This introduces denial of service or
even some dangerous loops.

Ddt provides an extensive analysis of the
presence of the name ``localhost'' in the database warning with several
severity levels about the possible problems that this practice can introduce.

\section{Conclusions}

The Ddt commands provide help for the administration of DNS zones. They can
assume an important role in the detection of several errors in zone configuration
files that otherwise would not be detected so quickly. Some of these errors
are very frequent and can introduce very dangerous and resource consuming
behaviors in the applications using the DNS or in the system itself.

There are still some open issues. Some of those issues are closely related
with the options taken by the bind software during the zone transfers.
An example of this is the problem of knowing if a glue record transferred with
the zone is or is not defined
in the transferred master file. We concluded that if the server from where the zone is pulled
has an authoritative answer for that host, then its IP address (glue
record) is always pulled with the remaining data of the zone, even if it is not
defined in the zone master file. This is due to the current implementation of the
bind software and will be corrected in future releases, we hope.

Other open issues are the analysis of which are the correct policies of mail
routing for a zone and which are the better SOA timers to use. Ddt has no
special treatment of those two issues. Mail routing is more or less a
policy issue if the related site is not fully interconnected with all parts
of the internet, or if there are mutual agreements for backup of mail
receipt. SOA timers are also closely related with the quality of the lines
available.

We have not experienced many difficulties with the fact that Ddt only works
with cached data (Ddt tests if a server is running authoritative for
a domain, otherwise only cached data is used).
Generally, zone updates are very occasional and, when analysing only one
zone, the probability of using outdated data is low. Administrators tend
to run the analysis just after having cached the zone file from the primary
server. In these situations, the test is run a few seconds after the caching took place.
This is not the case when we analyse extensively a significant part of the tree,
as for example all the sub-tree starting with the ``de.'' node. In these
cases, caching time is significant, and when the analysis starts, some cached
zones may not be consistent with the real ones any longer. However, these extensive analysis have,
generally, the purpose of getting statistics. As the management of the DNS tree is
decentralized, an administrator only deals directly with a small fraction
of the zones, mainly zones directly below the zone for which he or she is
directly responsible. Moreover, practice shows that, with the exception of name
servers, CNAMEs and MX pointers, DNS data has, in general, only local (in the same zone)
references.

During the implementation of Ddt we also took the option of ignoring TTL
values (``Time To Live'') associated with the cached data. Obviously,
Ddt only deals with authoritative data and it is up to its user
to garantee that he or she runs the analysis on the current version of the
zone database.

Finally, all the Ddt commands only analyze RRs of the Internet type (IN).
This restriction is due to the few weight of others types.  We decided
to wait until other types of records have a wider use.

Analysing european DNS data has been an exciting exercise providing us
an extensive set of examples of correct and incorrect practices, as well
more confidence  about the richness of European cultures. Networking
in such a disparate environment is a real challenge.

The Ddt package is public domain and available in the server {\em dec4pt.puug.pt}
by anonymous ftp under the name {\em pub/ddt.1.0beta}

\section{Acknowledgements}

Ddt has been developped in the Informatics Department of the Faculty of Sciences
of the University of Lisbon, in conjunction with ``PUUG -- The Portuguese Unix
Systems Users Group''.
This work has also received a grant from ``RCCN Network -- Portuguese
National R\&D Network''


\bibliography{ddt}

\bibliographystyle{plain}

\newpage
\tableofcontents
\listoffigures
\listoftables
\end{document}

